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(57) Abstract: A pen (12) or stylus-operated graphical user interface for a computer (10) or computing device, which includes a 
Q sensing surface (1 1 Thaving an atea corresponding to a data input field, the data input field being conditioned for hand entering and 
^ editing of graphical inpui symbols (13) , and user recognition software operative to analyze the graphical input symbols (13) and 

superimposing a display field of character dala (32) corresponding to the graphical input symbols on the data input field. 



10/530184 

Rec^d PCT/PTO G4 f\PR?nr\^ 

METHOD OF COMBE4XRG.PAJA.EOT 

SYMBOLS WITH DISPLAYED CHARACTER DATA 

FIELD 

The present invention relates to a method for combining data entry produced with 
a stylus on a sensing surface such as a computer touch screen or digitising tablet, with display of 
the character data corresponding to each haindwritten symbol. Handwriting recognition software 
is used to produce the character data corresponding to each symbol. 

BACKGROUND 

Systems with handwriting recognition include electronic notebooks and personal 
digital assistants (PDAs), which are portable computers incorporating a touch screen graphics 
display; and also non-portable computer workstations equipped with a digitising tablet and 
graphics display. Both types of systems have a pen input function when the user draws or writes 
with a stylus on the surface of the touch screen or digitising tablet. For handwritten data entry, 
such systems utilize a graphical user interface (GUI) presenting two spatially separate visual 
fields on the graphics display: first, a field where text characters are to be inserted by a text 
editing software prognon into a document (display field), usually showing a cursor to indicate 
the point of insertion for character data; and second, one or more fields (entry fields), where the 
user draws with the stylus to enter handwritten data. 

After recognition and conversion of the handwritten data, the resulting character 
data appear in the display field at the point of insertion indicated by the cursor. In a typical 
design, not only are the entry and display fields spatially separate, but also the position, size, 
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location* and other features of the character data bear little relation to the appearance of the 
original handwritten input. 

When the stylus is moved outside of an entry field, it typically operates as a 
5 pointing device to invoke other functions of the computer, such as editing text contained in the 
display field, and changing the insertion point in the display field. 

Typical prior methods of data entry with a stylus present the following difficulties 

to the user. 

10 1) visual attention must constantly be shifted between the entry and display 

fields; 

2) the stylus must be moved repeatedly between the display fields, to perform 
editing functions, and the entry fields, to continue entering handwritten data; 

3) the separate entry fields may use as much as one half of the available 
15 graphics display area on a small hand-held device such as a PDA, reducing the amount of other 
information that can be displayed; 

4) often, users must select the desired writing mode (characters, numbers, 
punctuation) and may forget which writing mode is currently active, or may enter the wrong type 
of handwritten symbol in an entry field; and 
20 5) in many systems each entry field accepts a single character only, which must 

be recognized before the system will accept further handwritten data. 

U.S. Patent No. 5,528,743 issued to Tou et al.describes a method of inputting 
information for the purpose of word processing. In word processing displayed text is formatted 
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into words, sentences and paragraphs. Tou et al. describes a number of calculations based on the 
spatial relationship of the position of newly entered graphical input symbols to the positions of 
paragraphs of formatted character data already existing on the display. Displayed character data 
are formatted into paragraphs, not necessarily superimposed on input symbols. For example, in 
5 Figure 3a of Tou et al. the handwritten symbol "three" entered near the right margin is displayed 
in Fig. 3b near the center of the line. The handwritten input below the existing character data in 
Fig. 3b is displayed as character data on the line above Fig. 3c, not in the position in which it was 
entered. 



10 European Patent Publication No. EP 0689124 (Canon) is concerned with 

recognizing handwritten symbols which arc gestures and distinguishing these from other 
handwritten input, and performing editing and other operations specified by the gestures. The 
Interaction described in the body of the patent, and in their claims, requires several steps with 
menu prompts for the user to execute a gesture action. In the 'Description of Related Art' Canon 

15 states that when handwritten characters are input to arbitary positions, recognition processing for 
the characters is performed and the recognition results are displayed at the positions where the 
handwriting input operation was performed. However, it is not disclosed whether a user must 
wait after writing each character until the system analyses and prints out that character or carries 
out the conunand operation represented by that character before writing the next character. It is 

20 also not disclosed whether or not the recognized input characters are superimposed on the 
handwritten input characters or are merely placed near them. 

U.S. Patent No. 5276794 issued to Lamb describes the handwritten entry into a 
single-line field based on the position where the handwriting is initiated, or in the field that is 
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active, even if some or all of the handwriting falls outside the display field. Character data are 
displayed left-justified in the single-line field, regardless of location of the handwritten input. 
For example, in Fig. 2a the four symbols "FRED" are entered in the right half of the input field, 
^but the recognition is displayed near the margin (see Fig. 2b). In addition, the recognition and 
5 display data are not immediate, but triggered by a time-out or other strategy to determine when 
the user has completed the input for a field. 

U.S. Patent No. 5,220,649 issue to Forcier describes a system in which 
handwritten input is displayed as ''digital ink'* (i.e., not automatically recognized and converted 
10 to character data) and editing mode is initiated with a timeout 

Accordingly, it is an object of the present invention to provide an improved means 
of data entry and editing by superimposing the input field and the display field on a GUI. It is a 



further object of the invention to provide an interface in which graphic symbols are entered by 
the user in an input field, and then are immediately replaced with the symbols' corresponding 
character data in approximately the same location. It is yet a further object of the invention to 
provide a means of correcting and editing character data without moving the stylus outside the 
5 input field. 

SUMMARY OF THE INVENTION 

According to the invention there is provided a pen or stylus-operable system for a 
computer or computing device, which includes a graphical user interface coupled to said 
computer and having a sensing surface, the sensing surface controlled by the graphical user 
interface and having means for hand entering and editing of graphical input symbols. 
Handwriting recognition software on the computer is operative to analyze the graphical input 
symbols one after another without interruption or delay and to superimpose a display field of 
character data corresponding to the graphical input symbols on the data input field. 

Advantageously, the sensing surface is a display surface. Alternatively, the 
sensing surface could be a tablet separate from the display surface. 

The handwriting recognition software also initiates an action based upon the 
20 graphical input symboL Preferably, the action is an editing mode wherein the pen or stylus 
contacts the sensing surface without moving for a predetermined minimum amount of time. 
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Symbol recognition of handwritten input is a default mode and editing mode is 

i 

"initiated with a timeout. 

Preferably movement of the pen, in predefined ways, without being removed from 
data input field, causes corresponding editing functions to be effected. 

The character data may be corrected and edited in the editing mode without 
moving a cursor for the pen or stylus outside the data input field of the sensing surface. 

. In another aspect of the invention there is provided a apparatus for combining 
data entry of handwritten symbols with displayed character data in a pen or stylus-operable 
graphical user interface for a computer or computing device, which includes means for recording 
and displaying handwritten graphical input symbols as they are entered on a data input field of a 
display surface; and handwriting recognition software for analysing continuously, without 
intermption or delay and automatically superimposing on the display field character data 
corresponding to the graphical input symbols. 

Preferably, the means for recording is a sensing surface operative to receive and 
record the graphical input symbols. The means for displaying is the display surface or, 
alternatively may be a part of the display surface. 



The handwriting recognition software may initiate an action based upon the 
graphical input symbol. The action may be an editing mode when the pen or stylus contacts the 

display surface for a predetermined minimum time without moving. 

J 

5 Movement of the pen in predefined ways, without being removed from the data 

input field, may cause conesponding editing functions to be effected. 



Character data may be conected and edited in the editing mode without moving 
the pen or stylus outside the data input field. 
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A pen or stylus operable system for a computer, comprising: 
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(a) a graphical user interface coupled to said computer (1 10, 1 1 2, 114) and having 
a sensing surface (1 1) said sensing surface having an area corresponding to a 
data input field (13), said sensing surface controlled by said graphical user 
interface and having means for hand entering and editing of gr^hical input 
symbols (31,32); 

(c) handwriting recognition software on said computer operative to analyze said 
graphical input symbols (31 , 32) one after another without intermption or 
delay and to automatically superimpose on and replace said graphical input 
symbols with a display field of character data (33) corresponding to said 
graphical input symbols (31» 32) on said data input field (13). 

2. An interface according to claim 1 , wherein said sensing surface ( 1 12) is a disftoy 



surface. 



3. 



An interface according to claim 1, wherein said sensing surface (1 12) is a tablet (1 1) 



separate from a display surface. 



4. 



An interface according to claim 1, wherein said handwriting recognition software also 



initiates an action based upon said graphical input symbol. 
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5. An interface according to claim I. wherein said handwriting recognition software 
initiates an editing mode when said pen or stylus (12) contacts said sensing surface (11) 
without moving for a predetermined nnnimum amount of time. 

6. An interface according to claim 5, wherein symbol recognition of handwritten input is 
a default mode and editing mode is initiated with a timeout. 

7. An interface according to claim 5, wherein movement of said pen (12), in predefined 
ways, while being held in continuous contact with the sensing surface, without being 
removed from said data input field, activates corresponding editing functions. 

8. An interface according to claim 7, wherein said character data is corrected and edited 
in said editing mode without moving a cursor for said pen or stylus (12) outside said data 
input field of said sensing surface (11). 

9. Apparams for combining data ratry of handwritten symbols with displayed character 
data in a pen or stylus-operable graphical user interface for a computer or computing device, 
comprising: 

* 

(a) means for recording and displaying handwritten gr^hical input symbols as 
they are entered on a data input field of a display surface; and 

(b) user recognition software for analyzing said graphical iiq>ut symbols one after 
another continuously without interruption or delay and automatically 



14 



iiupcnmposing on unci replacing said graphical inpul symbols (31, 33) wiih tt 
display field character data corresponding to said graphical input symbols. 

10. Apparatus according to claim 9, wherein said means for recording is a sensing surface 
(II) operative to receive and record the graphical inpul symbols. 

1 1 . Apparatus according to claim 10, wherein said means for displaying is said display 
surface (112). 

12. Apparatus according to claim 10, wherein said sensing surface (1 12) is at least part of 
said display surface (112). 

13. Apparatus according to claim 9, wherein said handwriting recognition software also 
initiates an action based upon said graphical input symbol. 

14. Apparatus according to claim 9, wherein said handwriting recognition software 
initiates an editing mode when said pen or stylus (12) contacts said display surface (1 12) for a 
predetermined minimum time without moving. 

15. Apparatus according to claim 14, wherein movement of said pen, without being 
removed from said data input field, in predefined ways, while held in continuous contact with 
the sensing surface, activates corresponding editing functions to be effected. 
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16. Apparatus according to claim IS, wherein character data is corrected and edited in 
said editing mode without moving a cursor for said pen or stylus outside sadd data input field. 
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16. Apparatus according to claim 15, wherein character data is corrected and edited in 
said editing mode without moving a cursor for said pen or stylus outside said data input field. 
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A pen or stylus-operated graphical user interface for a computer or computing 
device, which includes a sensing surface having an area corresponding to a data input field, the 
data input field being conditioned for hand entering and editing of graphical input symbols, and 
handwriting recognition software operative to analyze the graphical input symbols and 
superimposing a display field of character data corresponding to the graphical input symbols on 
the data input field. 
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FIG. 3 
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FIG. 4 
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FIG. 6 
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FIG. 7 
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FIG. 8 
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FIG. 10 



